Neural Network Training Research Articles

‘cito': an R package for training neural networks using ‘torch'

‘cito': an R package for training neural networks using ‘torch'

Typicality- and instance-dependent label noise-combating: a novel framework for simulating and combating real-world noisy labels for endoscopic polyp classification

Learning with noisy labels aims to train neural networks with noisy labels. Current models handle instance-independent label noise (IIN) well; however, they fall short with real-world noise. In medical image classification, atypical samples frequently receive incorrect labels, rendering instance-dependent label noise (IDN) an accurate representation of real-world scenarios. However, the current IDN approaches fail to consider the typicality of samples, which hampers their ability to address real-world label noise effectively. To alleviate the issues, we introduce typicality- and instance-dependent label noise (TIDN) to simulate real-world noise and establish a TIDN-combating framework to combat label noise. Specifically, we use the sample’s distance to decision boundaries in the feature space to represent typicality. The TIDN is then generated according to typicality. We establish a TIDN-attention module to combat label noise and learn the transition matrix from latent ground truth to the observed noisy labels. A recursive algorithm that enables the network to make correct predictions with corrections from the learned transition matrix is proposed. Our experiments demonstrate that the TIDN simulates real-world noise more closely than the existing IIN and IDN. Furthermore, the TIDN-combating framework demonstrates superior classification performance when training with simulated TIDN and actual real-world noise.

Neural Technologies for Objects Classification with Mobile Applications

This paper is related to the study of the features of the neural technologies’ application, in particular, ResNet neural networks for the classification of objects in photographs. The work aims to increase the accuracy of recognition and classification of objects in photographs by using various models of the ResNet neural network. The paper analyzes the features of the application of the corresponding models in comparison with other architectures of deep neural networks and evaluates their efficiency and accuracy in the classification of objects in photographs. The process of data formation for training neural networks, their processing and sorting is described. A web application and a mobile application for recognizing and classifying objects in a photo were also developed. A system for classifying objects, in particular airplanes in photographs, was developed using neural network technologies. It gives a recognition and classification accuracy of about 95%. Research results of ResNet models are of great practical importance, as they can improve the classification accuracy of various images. Features of ResNet, such as the use of skip connections or residual connections, make it effective in the relevant tasks. The results of the study will help to implement ResNet in various fields, including medicine, automatic pattern recognition and other areas where the classification of objects in photographs is an important task.

Neural lasso: a unifying approach of lasso and neural networks

AbstractIn recent years, there has been a growing interest in establishing bridges between statistics and neural networks. This article focuses on the adaptation of the widely used lasso algorithm within the context of neural networks. To accomplish this, the network configuration is first designed. After that, in order to estimate the network weights, three optimization algorithms are considered. The first one, called standard neural lasso, employs the conventional procedure for training neural networks. The second optimization algorithm, termed restricted neural lasso, mimics traditional lasso to establish a connection between statistics and machine learning. Finally, a third optimization algorithm, called voting neural lasso was developed. Voting neural lasso offers a novel way of estimating weights by considers the significance of variables across the cross-validation scenarios. Results showed that the conventional approach of training neural networks resulted in a lower performance when the validation set is not sufficiently representative. It was also observed that restricted neural lasso and the traditional lasso obtained equivalent results, which shows the convergence of the neural technique with the statistical one. Finally, the developed voting neural lasso algorithm outperformed the traditional lasso. These results were obtained across diverse training sets, encompassing observations ranging from as few as 47 to as many as 4000, with the number of predictors varying from 9 to 200.

Data Pruning-enabled High Performance and Reliable Graph Neural Network Training on ReRAM-based Processing-in-Memory Accelerators

Graph Neural Networks (GNNs) have achieved remarkable accuracy in cognitive tasks such as predictive analytics on graph-structured data. Hence, they have become very popular in diverse real-world applications. However, GNN training with large real-world graph datasets in edge-computing scenarios is both memory- and compute-intensive. Traditional computing platforms such as CPUs and GPUs do not provide the energy efficiency and low latency required in edge intelligence applications due to their limited memory bandwidth. Resistive random-access memory (ReRAM)-based processing-in-memory (PIM) architectures have been proposed as suitable candidates for accelerating AI applications at the edge, including GNN training. However, ReRAM-based PIM architectures suffer from low reliability due to their limited endurance, and low performance when they are used for GNN training in real-world scenarios with large graphs. In this work, we propose a learning-for-data-pruning framework, which leverages a trained Binary Graph Classifier (BGC) to reduce the size of the input data graph by pruning subgraphs early in the training process to accelerate the GNN training process on ReRAM-based architectures. The proposed light-weight BGC model reduces the amount of redundant information in input graph(s) to speed up the overall training process, improves the reliability of the ReRAM-based PIM accelerator, and reduces the overall training cost. This enables fast, energy-efficient, and reliable GNN training on ReRAM-based architectures. Our experimental results demonstrate that using this learning for data pruning framework, we can accelerate GNN training and improve the reliability of ReRAM-based PIM architectures by up to 1.6 ×, and reduce the overall training cost by 100 × compared to state-of-the-art data pruning techniques.

Random feedback alignment algorithms to train neural networks: why do they align?

Feedback alignment algorithms are an alternative to backpropagation to train neural networks, whereby some of the partial derivatives that are required to compute the gradient are replaced by random terms. This essentially transforms the update rule into a random walk in weight space. Surprisingly, learning still works with those algorithms, including training of deep neural networks. The performance of FA is generally attributed to an alignment of the update of the random walker with the true gradient—the eponymous gradient alignment—which drives an approximate gradient descent. The mechanism that leads to this alignment remains unclear, however. In this paper, we use mathematical reasoning and simulations to investigate gradient alignment. We observe that the feedback alignment update rule has fixed points, which correspond to extrema of the loss function. We show that gradient alignment is a stability criterion for those fixed points. It is only a necessary criterion for algorithm performance. Experimentally, we demonstrate that high levels of gradient alignment can lead to poor algorithm performance and that the alignment is not always driving the gradient descent.

Intelligent crack identification method for high‐rise buildings aided by synthetic environments

SummaryCracks can develop in high‐rise buildings because of long‐term environmental changes and extreme loading events such as strong winds or earthquakes. Although deep learning‐based identification methods can efficiently identify cracks, the accuracy of crack identification in high‐rise buildings needs to be improved due to the lack of crack datasets specifically related to high‐rise structures. Moreover, the number of available images of cracks in high‐rise is limited. To this end, this paper establishes an intelligent crack identification method based on a photorealistic synthetic modeling technique. First, a computer graphics (CG) model of a high‐rise building with assumed damage is constructed. Subsequently, the CG model is utilized to generate a dataset that includes photorealistic images of the high‐rise building as well as corresponding labels for various components and types of damage. The generated dataset is then used to train a DeepLabv3 + neural network for structural component and damage identification, followed by validation by employing images of both synthetic and full‐scale high‐rise buildings. The trained network can accurately identify different components in images of the full‐scale, high‐rise building and identify cracks that are intentionally synthesized in those images. The results show that the synthetic dataset generated by the CG model not only allows for fast and efficient labeling for the purpose of neural network training but also outperforms methods that do not consider any application‐specific context in crack identification.

Approximate Dynamic Programming for Control of Wave Energy Converters With Implementation and Validation on a Point Absorber Prototype

It is recognized that the application of constrained optimal control for wave energy converters (WECs), represented by model predictive control (MPC), is hindered by its computation burden resulting from online optimization, especially when the model contains nonlinearities. In this article, a novel control solution based on approximate dynamic programming (ADP) is proposed, in which an explicit controller is solved by model sampling and neural network training. In this way, the major computation load is moved offline to enable fast online execution, and the need for wave prediction is eliminated. The effectiveness of ADP is verified by a simulation case study based on a nonlinear WEC system. Using nonlinear MPC as the performance benchmark, over 90% energy efficiency is achieved by ADP under various sea states with constraint satisfaction and dramatically faster computation. The ADP controller is further experimentally validated on a newly constructed WEC prototype through wave tank testing, where successful implementation and satisfactory performance are demonstrated.

Why do we need to employ exemplars in moral education? Insights from recent advances in research on artificial intelligence

ABSTRACT In this paper, I examine why moral exemplars are useful and even necessary in moral education despite several critiques. To support my point, I review recent AI research demonstrating that exemplar-based learning is superior to rule-based learning in model performance in training neural networks, such as large language models. I particularly focus on why education aiming at promoting the development of multifaceted moral functioning can be done effectively by using exemplars, which is like exemplar-based learning in AI model training. Furthermore, I discuss the potential limitations and issues related to exemplar-applied moral education with findings from recent AI research raising concerns about model biases and toxic outcomes. I propose ways to address the concerns regarding employing moral exemplars as well. As remedies, I suggest that autonomy-supporting deliberative and reflective learning should be utilized. Furthermore, based on the discussion, I examine how macroscopic socio-cultural aspects influence the effectiveness of exemplar-applied education.

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

Designing and Training of Lightweight Neural Networks on Edge Devices Using Early Halting in Knowledge Distillation

Brave the Wind and the Waves: Discovering Robust and Generalizable Graph Lottery Tickets.

The training and inference of Graph Neural Networks (GNNs) are costly when scaling up to large-scale graphs. Graph Lottery Ticket (GLT) has presented the first attempt to accelerate GNN inference on large-scale graphs by jointly pruning the graph structure and the model weights. Though promising, GLT encounters robustness and generalization issues when deployed in real-world scenarios, which are also long-standing and critical problems in deep learning ideology. In real-world scenarios, the distribution of unseen test data is typically diverse. We attribute the failures on out-of-distribution (OOD) data to the incapability of discerning causal patterns, which remain stable amidst distribution shifts. In traditional spase graph learning, the model performance deteriorates dramatically as the graph/network sparsity exceeds a certain high level. Worse still, the pruned GNNs are hard to generalize to unseen graph data due to limited training set at hand. To tackle these issues, we propose the Resilient Graph Lottery Ticket (RGLT) to find more robust and generalizable GLT in GNNs. Concretely, we reactivate a fraction of weights/edges by instantaneous gradient information at each pruning point. After sufficient pruning, we conduct environmental interventions to extrapolate potential test distribution. Finally, we perform last several rounds of model averages to further improve generalization. We provide multiple examples and theoretical analyses that underpin the universality and reliability of our proposal. Further, RGLT has been experimentally verified across various independent identically distributed (IID) and out-of-distribution (OOD) graph benchmarks.

The ZTF Source Classification Project. III. A Catalog of Variable Sources

The classification of variable objects provides insight into a wide variety of astrophysics ranging from stellar interiors to galactic nuclei. The Zwicky Transient Facility (ZTF) provides time-series observations that record the variability of more than a billion sources. The scale of these data necessitates automated approaches to make a thorough analysis. Building on previous work, this paper reports the results of the ZTF Source Classification Project (SCoPe), which trains neural network and XGBoost (XGB) machine-learning (ML) algorithms to perform dichotomous classification of variable ZTF sources using a manually constructed training set containing 170,632 light curves. We find that several classifiers achieve high precision and recall scores, suggesting the reliability of their predictions for 209,991,147 light curves across 77 ZTF fields. We also identify the most important features for XGB classification and compare the performance of the two ML algorithms, finding a pattern of higher precision among XGB classifiers. The resulting classification catalog is available to the public, and the software developed for SCoPe is open source and adaptable to future time-domain surveys.

Arterial Input Function (AIF) Correction Using AIF Plus Tissue Inputs with a Bi-LSTM Network

Background: The arterial input function (AIF) is vital for myocardial blood flow quantification in cardiac MRI to indicate the input time–concentration curve of a contrast agent. Inaccurate AIFs can significantly affect perfusion quantification. Purpose: When only saturated and biased AIFs are measured, this work investigates multiple ways of leveraging tissue curve information, including using AIF + tissue curves as inputs and optimizing the loss function for deep neural network training. Methods: Simulated data were generated using a 12-parameter AIF mathematical model for the AIF. Tissue curves were created from true AIFs combined with compartment-model parameters from a random distribution. Using Bloch simulations, a dictionary was constructed for a saturation-recovery 3D radial stack-of-stars sequence, accounting for deviations such as flip angle, T2* effects, and residual longitudinal magnetization after the saturation. A preliminary simulation study established the optimal tissue curve number using a bidirectional long short-term memory (Bi-LSTM) network with just AIF loss. Further optimization of the loss function involves comparing just AIF loss, AIF with compartment-model-based parameter loss, and AIF with compartment-model tissue loss. The optimized network was examined with both simulation and hybrid data, which included in vivo 3D stack-of-star datasets for testing. The AIF peak value accuracy and ktrans results were assessed. Results: Increasing the number of tissue curves can be beneficial when added tissue curves can provide extra information. Using just the AIF loss outperforms the other two proposed losses, including adding either a compartment-model-based tissue loss or a compartment-model parameter loss to the AIF loss. With the simulated data, the Bi-LSTM network reduced the AIF peak error from −23.6 ± 24.4% of the AIF using the dictionary method to 0.2 ± 7.2% (AIF input only) and 0.3 ± 2.5% (AIF + ten tissue curve inputs) of the network AIF. The corresponding ktrans error was reduced from −13.5 ± 8.8% to −0.6 ± 6.6% and 0.3 ± 2.1%. With the hybrid data (simulated data for training; in vivo data for testing), the AIF peak error was 15.0 ± 5.3% and the corresponding ktrans error was 20.7 ± 11.6% for the AIF using the dictionary method. The hybrid data revealed that using the AIF + tissue inputs reduced errors, with peak error (1.3 ± 11.1%) and ktrans error (−2.4 ± 6.7%). Conclusions: Integrating tissue curves with AIF curves into network inputs improves the precision of AI-driven AIF corrections. This result was seen both with simulated data and with applying the network trained only on simulated data to a limited in vivo test dataset.

A Linear Interpolation and Curvature-Controlled Gradient Optimization Strategy Based on Adam

The Adam algorithm is a widely used optimizer for neural network training due to efficient convergence speed. The algorithm is prone to unstable learning rate and performance degradation on some models. To solve these problems, in this paper, an improved algorithm named Linear Curvature Momentum Adam (LCMAdam) is proposed, which introduces curvature-controlled gradient and linear interpolation strategies. The curvature-controlled gradient can make the gradient update smoother, and the linear interpolation technique can adaptively adjust the size of the learning rate according to the characteristics of the curve during the training process so that it can find the exact value faster, which improves the efficiency and robustness of training. The experimental results show that the LCMAdam algorithm achieves 98.49% accuracy on the MNIST dataset, 75.20% on the CIFAR10 dataset, and 76.80% on the Stomach dataset, which is more difficult to recognize medical images. The LCMAdam optimizer achieves significant performance gains on a variety of neural network structures and tasks, proving its effectiveness and utility in the field of deep learning.

Isocentric fixed angle irradiation-based DRR: a novel approach to enhance x-ray and CT image registration.

Digitally Reconstructed Radiography (DRR) plays an important role in the registration of intraoperative X-ray and preoperative CT images. However, existing DRR algorithms often neglect the critical Isocentric Fixed Angle Irradiation (IFAI) principle in C-arm imaging, resulting in inaccurate simulation of X-ray images. This limitation degrades registration algorithms relying on DRR image libraries or employing DRR images (DRRs) to train neural network models. To address this issue, we propose a novel IFAI-based DRR method that accurately captures the true projection transformation during X-ray imaging of the human body.
Approach. By strictly adhering to the IFAI principle and utilizing known parameters from intraoperative X-ray images paired with CT scans, our method successfully simulates the real projection transformation and generates DRRs that closely resemble actual X-ray images.
Main result. Experimental results validate the effectiveness of our IFAI-based DRR method by successfully registering intraoperative X-ray images with preoperative CT images from multiple patients who underwent thoracic endovascular aortic procedures.
Significance. The proposed IFAI-based DRR method enhances the quality of DRR images, significantly accelerates the construction of DRR image libraries, and thereby improves the performance of X-ray and CT image registration. Additionally, the method has the generality of registering CT and X-ray images generated by large C-arm devices.&#xD.

Artificial neural network and discrete wavelet transform for inter-turn short circuit and broken rotor bars faults diagnosis under various operating conditions

Introduction. This work presents a methodology for detecting inter-turn short circuit (ITSC) and broken rotor bars (BRB) fault in variable speed induction machine controlled by field oriented control. If any of these faults are not detected at an early stage, it may cause an unexpected shutdown of the industrial processes and significant financial losses. Purpose. For these reasons, it is important to develop a new diagnostic system to detect in a precautionary way the ITSC and BRB at various load condition. We propose the application of discrete wavelet transform to overcome the limitation of traditional technique for no-stationary signals. The novelty of the work consists in developing a diagnosis system that combines the advantages of both the discrete wavelet transform (DWT) and artificial neural network (ANN) to identify and diagnose defects, related to both ITSC and BRB faults. Methods. The suggested method involves analyzing the electromagnetic torque signal using DWT to calculate the stored energy at each level of decomposition. Then, this energy is applied to train neural network classifier. The accuracy of ANN based on DWT, was improved by testing different orthogonal wavelet functions on simulated signal. The selection process identified 5 pertinent wavelet energies, concluding that, Daubechies44 (db44) is the best suitable mother wavelet function for effectively detecting and classifying failures in machines. Results. We applied numerical simulations by MATLAB/Simulink software to demonstrate the validity of the suggested techniques in a closed loop induction motor drive. The obtained results prove that this method can identify and classify these types of faults under various loads of the machine. References 31, table 1, figures 9.

Blood Cell Attribute Classification Algorithm Based on Partial Label Learning

Hematological morphology examinations, essential for diagnosing blood disorders, increasingly utilize deep learning. Blood cell classification, determined by combinations of cell attributes, is complicated by the complex relationships and subtle differences among the attributes, resulting in significant time and cost penalties. This study introduces the Partial Label Learning for Blood Cell Classification (P4BC) strategy, a method that trains neural networks using the blood cell attribute labeling data of weak annotations. Using morphological knowledge, we predefined candidate label sets for the blood cell attributes to blend this knowledge with deep learning. This improves the model’s prediction accuracy and interpretability in classifying attributes. This method effectively combines morphological knowledge with deep learning, an approach we refer to as knowledge alignment. It results in an 8.66% increase in attribute recognition accuracy and a 1.09% improvement in matching predictions to the candidate label sets, compared to the original method. These results confirm our method’s ability to grasp the characteristic information of blood cell attributes, enhancing the model interpretability and achieving knowledge alignment between hematological morphology and deep learning. Our algorithm ensures attribute classification accuracy and shows excellent cell category classification, highlighting its wide application potential and practical value in blood cell category classification.

Integration of Artificial Intelligence for Enhanced Coordination of DOCR Protection in Distributed Generation Systems

Distributed generation (DG) is an approach that involves adding decentralized power generation within a distribution network. Distributed generation systems can reduce transmission losses, increase the reliability of energy supply, minimize carbon emissions, and enable the active participation of consumers in energy production. However, with the increase in distributed generation, electric power systems face new challenges in maintaining operational reliability and safety. Disruptions such as short circuits or overcurrent can occur in the system, and appropriate protective responses are required to protect the power grid from more significant damage. The addition of DG also causes the short circuit current to vary and results in system protection coordination having to be redone. Carrying out coordination will take a long time. This research uses modeling and simulation of a distributed generation system with various operating conditions and works adaptively according to changes in the system due to the addition of DG. The results obtained from the simulation are used in neural network training to study the relationship patterns between directional overcurrent relays (DOCR) parameters and system operating conditions. The backpropagation algorithm is used in the Artificial Neural Network (ANN) training process. The training process utilizes the maximum Short Circuit Current (ISC) input obtained through generation, fault location, and fault type. Time Dial Setting (TDS) and Ipickup values are used as ANN training targets. After testing, the results obtained are in accordance with the target data. The efficacy of this method is further demonstrated through ETAP simulations, which confirm that ANN is a suitable approach for modeling adaptive and optimal relay coordination systems.

Detection of Skin Disease by using Image Processing

There are many types and forms of skin diseases and have many causes, including internal ones related to hormones and body glands such as acne, or external related to air pollution or sensitivity to sunlight such as rashes. The skin disease may be contagious, such as scabies and lice, or it may not be contagious, such as drug allergy and rosacea, or it may be chronic, such as psoriasis and atopic eczema, or it may be rare, such as sweet syndrome and ofuji disease. The outlook of society, in general, reduces the importance of skin disease when it appears and it is preferable not to visit the doctor. Skin diseases are a major and worrying problem in societies due to their physical and psychological effects on patients. Detecting skin diseases at an early stage has an important role in treatment. The process of diagnosing and treating skin injury is related to the skill and experience of the specialist doctor. skin disease prediction by using a neural network based on image analysis. Diagnosis and prediction of skin disease take a very long process because it requires a patient’s history, physical examination, and proper laboratory diagnostic tests. st of such diagnosis is still limited and very expensive. So, image processing techniques help to build automated screening system for dermatology at an initial stage. The extraction of features plays a key role in helping to classify skin disease. proach works on the inputs of a color image. Then resize the of the image to extract features using pre trained convolutional neural network. After that classified feature using Multiclass SVM. Finally, the results are shown to the user, including the type.

Neural Network Modeling of a Stereolithography Printed Mesostructure

Abstract This paper addresses the scarcity of comprehensive studies on the collective impact of various parametric lattice designs on mesostructure functionality. Focusing on optimizing the energy absorption of a serpentine mesostructure made using stereolithography, this research leverages a feedforward neural network to explore the interplay between line width, number of turns, and material properties on the energy absorbed by the structure. Compression simulations using a finite element model, covering a range of configurations, provided the dataset for neural network training. The resulting network was used to probe correlations between geometric variables, material, and energy absorption. Additionally, a neural network sensitivity analysis explored the impact of hidden layers and number of neurons on the network's performance, demonstrating the network's robustness. The optimized mesostructure configuration, identified by the neural network, maximized energy absorption. Using foundational mechanics of materials concepts, the discussion explains how the geometry and material of the cellular mesostructure affect structural stiffness.